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Capsule  Description 


Software  safety  involves  ensuring  that  software  will 
execute  within  a  system  context  without  resulting  in 
unacceptable  risk.  Building  safety-critical  software 
requires  special  procedures  to  be  used  in  all  phases 
of  the  software  development  process.  This  module 
introduces  the  problems  involved  in  building  such 
software  along  with  the  procedures  that  can  be  used 
to  enhance  the  safety  of  the  resulting  software  prod¬ 
uct. 


Philosophy 


Software  safety  has  been  an  important  topic  for 
many  years  in  defense  and  aerospace  systems.  Re¬ 
cently  it  has  become  even  more  important  for  soft¬ 
ware  engineers  to  study  this  topic  as  computers  are 
increasingly  used  to  monitor  and  control  safety- 
critical  devices  and  processes  in  such  disparate  areas 
as  medicine,  transportation,  energy,  manufacturing, 
etc.  Unfortunately,  few  software  engineers  are 
trained  in  this  area.  The  purpose  of  this  module  is  to 
introduce  the  problems  involved  in  building  safety- 
critical  systems  and  to  provide  instruction  on  some 
of  the  procedures  and  techniques  that  can  be  used  to 
solve  these  problems.  There  is  a  desperate  need  for 
software  engineers  with  this  training,  yet  few  classes 
currently  exist. 

The  module  could  be  used  as  part  of  another  course 
or  as  a  complete  course  on  its  own.  A  survey  paper 
is  available  to  supplement  a  short  presentation,  while 
a  book  is  planned  to  support  a  more  extensive  pres¬ 
entation  of  the  subject  matter. 

The  module  includes  both  introductory  material  and 
in-depth  presentation  of  detailed  analysis  techniques. 
Because  software  engineers  building  embedded  sys¬ 
tems  will  need  to  interact  with  system  engineers,  the 
module  includes  not  only  a  discussion  of  software 


engineering  techniques  but  also  an  introduction  to 
some  system  safety  engineering  procedures  with 
which  they  will  need  to  be  familiar. 


Objectives 


The  goal  of  the  module  is  to  equip  software  engi¬ 
neers  with  the  extra  knowledge  and  skills  necessary 
to  participate  in  a  safety-critical  software  develop¬ 
ment  project.  The  positions  for  which  they  are 
qualified  will  depend  upon  the  length  of  time  spent 
on  the  module  and  depth  of  presentation.  All  soft¬ 
ware  engineers  should  have  some  appreciation  of  the 
problems  involved  in  such  projects.  Those  who  will 
participate  in  these  projects  but  not  be  responsible 
for  safety  should,  in  addition,  have  some  exposure  to 
the  analysis  techniques  and  be  able  to  implement  the 
design  techniques.  Software  engineers  who  arc 
responsible  for  software  safety  must,  in  addition,  be 
able  to  carry  out  required  analysis  and  verification 
activities.  The  parts  of  the  module  taught  and  the 
depth  with  which  each  is  presented  will  depend  upon 
the  objectives  of  the  individual  instructor. 


Prerequisite  Knowledge 


There  are  no  explicit  prerequisites  beyond  a  basic 
introduction  to  software  engineering.  In-depth 
coverage  of  such  topics  as  software  safety  design 
techniques  and  risk  assessment  and  measurement 
will  require  some  prerequisite  knowledge  depending 
on  how  much  background  material  the  instructor 
wants  to  supply  while  teaching  the  course. 
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Module  Content 


Outline 


I.  What  is  Software  Safety 

1.  Introduction 

2.  What  is  Safety  and  Safety  Engineering 

3.  Why  is  There  a  Safety  Problem  with  Software 

4.  Relationship  of  Safety  with  Other  Software 
Qualities 

II.  Introduction  to  System  Safety 

1.  Motivation 

2.  Hazard  Analysis 

3.  Hazard  Control 

III.  Management  of  Safety-Critical  Software  Projects 

1.  Importance  in  Achieving  Safety 

2.  Responsibilities  of  Software  Management 

3.  Duties  of  Software  Safety  Group 

4.  Management  Structure 

IV.  Software  Safety  Modeling  and  Analysis 

1.  System  Hazard  Analysis 

2.  Software  Hazard  Analysis 

3.  Software  Safety  Requirements 

V.  Software  Design  Concepts  to  Enhance  Safety 

1 .  Why  the  Need  for  Runtime  Measures 

2.  Hazard  Prevention  vs.  Hazard  Control 

3.  Hazard  Prevention 

4.  Hazard  Detection  and  Treatment 

VI.  Verification  and  Certification  of  Safety 

VII.  Assessment  of  Safety 

1.  Introduction  to  Quantitative  Risk  Assessment 

2.  Probabilistic  Risk  Assessment 

3.  Software  Reliability  and  Safety 

VIII.  Man/Machine  Interface  Considerations 

IX.  Miscellaneous  Issues 


Annotated  Outline _ 

Most  material  directly  supporting  this  module  can  be 
found  in  [Leveson86].  Specific  references  are  made 
to  other  sources  where  appropriate. 


I.  What  is  Software  Safety 

1.  Introduction 

Safety  problems  arise  with  the  introduction  of  com¬ 
puters  into  safety-critical  systems.  An  embedded 
computer  system  is  one  in  which  the  computer  and 
its  software  form  part  of  some  larger  system,  usually 
electromechanical,  such  as  a  ship,  aircraft,  missile, 
spacecraft,  processing  plant,  or  rapid  transit  system. 
The  embedded  system  m-iy  be  physically  or  only 
logically  incorporated  into  the  larger  system.  Em¬ 
bedded  systems  are  usually  real-time  systems  that 
must  be  capable  of  responding  to  input  data  by  pro¬ 
ducing  control  signals  within  critical  real  time  limits. 
Basic  concepts  of  embedded  systems  should  be 
presented,  including  open  and  closed  loops,  and  the 
role  of  computers  and  humans  within  these  loops. 
An  introduction  to  these  ideas  can  be  found  in 
[AllworthSI,  Kopetz79]. 

The  instructor  may  want  to  present  some  examples 
of  accidents  with  computer  software  as  a  contribut¬ 
ing  factor.  The  examples  should  be  chosen  to  il¬ 
lustrate  the  various  ways  software  can  interact  with 
other  factors  to  lead  to  accidents.  Examples  can  be 
found  in  [Leveson86]  and  [NeumannBS]. 

The  problem  is  always  one  of  trading  off  benefits  vs. 
risks.  The  reasons  for  using  computers  in  safety- 
critical  systems — versatility,  power,  performance, 
efficiency — must  be  compared  against  the  added 
risks  of  inability  to  provide  correct  software.  Even 
where  computers  can  improve  safety,  it  is  not  clear 
they  will,  because  technological  improvements  often 
allow  running  greater  risks. 

2.  What  is  Safety  and  Safety  Engineering 

Safety  may  be  defined  as  "freedom  from  those  con¬ 
ditions  that  can  cause  death,  injury,  occupational  ill¬ 
ness,  or  damage  to  or  loss  of  equipment  or 
property."  But  safety  is  a  relative  concept  Nothing 
is  absolutely  safe  under  all  conditions.  Provide 
some  examples  and  then  get  students  to  provide 
some.  Discuss  typical  tradeoffs.  Describe  risk 
elimination  vs.  risk  displacement  Safety  therefore 
involves  an  attempt  to  provide  acceptable  risk. 

Provide  definitions  of  relevant  terms  including  sys¬ 
tem,  software  safety,  risk,  hazard,  accident,  mishap. 

Provide  a  brief  definition  of  system  safety  engineer¬ 
ing  and  its  history.  Describe  characteristics  of  acci¬ 
dents:  multifactorial,  occur  in  interfaces,  related  to 
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complexity  and  coupling.  This  is  a  good  place  to 
provide  a  detailed  description  of  a  complex  accident 
showing  how  these  factors  were  involved.  Ex¬ 
amples  can  be  found  in  [Perrow84], 

3.  Why  is  There  a  Safety  Problem  with  Software 

Safety  is  not  just  a  matter  of  increasing  reliability. 
We  are  currently  unable  to  achieve  ultra-high 
reliability  in  software  (explain  how  reliability  is 
specified  and  defined,  hardware  reliability  vs.  Soft¬ 
ware  reliability).  Hardware  reliability  techniques 
are  aimed  toward  random  failures.  Software  sys¬ 
tems  often  involve  increased  coupling  and  com¬ 
plexity.  Extensive  reuse  of  certified  software  is 
presently  infeasible.  Exhaustive  testing  and  verifi¬ 
cation  is  impractical  for  most  software.  Operating 
conditions  often  differ  from  test  conditions.  There  is 
no  way  to  guarantee  that  simulations  are  accurate. 
Assumptions  must  be  made  about  the  controlled 
process  and  its  environment.  The  problem  is 
amplified  when  writing  software  for  hardware  that  is 
new  or  does  not  yet  exist.  There  are  great  dif¬ 
ficulties  in  writing  correct  software  requirements. 
Software  fault  tolerance  techniques  are  limited  in  ef¬ 
fectiveness. 

4.  Relationship  of  Safety  with  Other  Software 
Qualities 

Discuss  the  relationship  of  safety  with  reliability, 
availability,  security.  All  can  be  put  under  a  general 
label  of  dependability  [Laprie82], 

II.  Introduction  to  System  Safety 

1.  Motivation 

It  is  important  for  software  engineers  to  understand 
something  about  system  safety  for  two  reasons: 

•  Software  is  just  one  part  of  the  system;  in  fact, 
software  is  only  unsafe  when  operating  within 
a  hazardous  system  context  Software  engi¬ 
neers  must  work  closely  with  safety  and  sys¬ 
tem  engineers  to  make  the  entire  system  safe. 
This  requires  that  software  engineers  under¬ 
stand  basic  system  safety  techniques. 

•  Software  often  replaces  standard  hardware 
safety  devices  such  as  interlocks.  Unless  the 
software  includes  the  software-equivalent  de¬ 
vices,  safety  will  be  compromised. 

2.  Hazard  Analysis 

Describe  the  goals  and  techniques  involved  in  the 
following  analyses  using  examples  to  illustrate 
them:  preliminary  hazard  analysis,  subsystem  haz¬ 
ard  analysis,  system  hazard  analysis,  operating  and 
support  hazard  analysis. 

3.  Hazard  Control 

The  goal  is  to  eliminate  hazards  or  to  minimize  their 


occurrence  and/or  severity  (effects).  Present  the  or¬ 
der  of  precedence  for  applying  safety  design  meas¬ 
ures  along  with  examples  of  each  in  hardware: 

•  design  for  mrnnsic  safety 

•  design  to  prevent  or  minimize  the  occurrence 
of  hazards  (e.g.,  monitoring,  automatic  con¬ 
trol,  lockouts,  lockins,  interlocks) 

•  design  to  control  hazard  if  it  occurs  using  au¬ 
tomatic  safety  devices  (e.g.,  hazard  detection, 
fail-safe  designs,  damage  control,  contain¬ 
ment,  isolation) 

•  provide  warning  devices,  procedures,  and 
training  to  help  personnel  react  to  hazard. 

III.  Management  of  Safety-Critical  Software  Projects 

1.  Importance  in  Achieving  Safety 

Degree  of  safety  achieved  is  directly  related  to  the 
amount  of  management  emphasis.  Management  al¬ 
ways  needs  to  prioritize  conflicting  or  costly  goals. 

2.  Responsibilities  of  Software  Management 

Discuss  the  implications  of  setting  policy  and  defin¬ 
ing  goals,  delegating  and  assigning  responsibility  for 
safety,  granting  authority,  fixing  accountability,  and 
delineating  lines  of  authority,  coordination,  and  ad¬ 
ministration. 

3.  Duties  of  Software  Safety  Group 

Discuss  the  duties  of  the  software  safety  group,  in¬ 
cluding  the  following: 

•  Participation  in  planning  the  software  aspects 
of  the  system  safety  plan.  Describe  what  such 
a  plan  is  and  provide  an  example.  Requires 
that  software  group  interact  closely  with  sys¬ 
tem  safety  group. 

•  Overall  responsibility  for  software  safety  anal¬ 
yses. 

•  Participation  with  delegated  software  safety 
responsibility  in  all  design  reviews  and  con¬ 
figuration  board  activities. 

•  Establishing  and  overseeing  audit  trails  for  all 
identified  software  hazards.  This  may  involve 
merely  responsibility  for  input  to  system  haz¬ 
ard  auditing  for  software  hazards. 

•  All  necessary  interfacing  with  the  system 
safety  group. 

•  General  participation  or  auditing  of  all  aspects 
of  software  development  activities  to  ensure 
that  software  hazards  are  minimized  and  con¬ 
troller  to  an  acceptable  degree. 

•  Production  of  any  documentation  of  the  safety 
aspects  of  the  software  which  are  needed  for 
certification  or  licensing  of  the  system. 

4.  Management  Structure 
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Discuss  the  position  of  the  system  safety  manage¬ 
ment  within  the  overall  management  structure  of 
typical  engineering  projects  and  where  the  software 
safety  group  should  be  inserted. 

IV.  Software  Safety  Modeling  and  Analysis 

1.  System  Hazard  Analysis 

The  goal  is  to  show  the  system  is  safe  if  it  operates 
as  intended  and  to  show  it  is  safe  in  the  presence  of 
faults.  System  safety  techniques  try  to  show  that  no 
single  fault  causes  a  hazard  and  that  hazards  from 
sequences  of  faults  are  sufficiently  remote.  The  lat¬ 
ter  approaches  the  impossible  if  an  attempt  is  made 
to  combine  all  possible  failures  in  all  possible  se¬ 
quences  and  to  analyze  the  output.  Instead,  system 
safety  approaches  often  involve  techniques  that  first 
define  what  is  hazardous  and  then  work  backward  to 
find  all  combinations  of  faults  that  could  produce 
that  event. 

2.  Software  Hazard  Analysis 

SHA  starts  from  the  system  hazard  analysis  and 
identifies  software  hazards  including  both  operating 
and  failure  modes,  single  and  multiple  failure  se¬ 
quences.  The  purpose  is  to  identify  the  safety- 
critical  areas  of  the  software  for  further  attention. 

3.  Software  Safety  Requirements 

Using  examples,  differentiate  between  mission  and 
other  requirements,  including  safety.  Software 
safety  requirements  are  derived  from  the  software 
hazard  analysis  and  system  preliminary  hazard  anal¬ 
ysis.  They  should  include  the  requirements  for  de¬ 
tecting,  eliminating,  and  controlling  hazards  and  for 
limiting  damage  in  case  of  an  accident;  the  ways  in 
which  the  software  and  system  can  fail  safely;  and 
the  extent  to  which  failure  is  tolerable.  Stress  the 
fact  that  safety  requirements  may  conflict  with  other 
requirements,  and  these  conflicts  must  be  deter¬ 
mined  and  resolved  before  software  can  be  built 

V.  Software  Design  Concepts  to  Enhance  Safety 

1.  Why  the  Need  for  Runtime  Measures 

Verification  and  analysis  is  not  enough  because  the 
techniques  are  so  complex  as  to  be  error-prone 
themselves,  the  cost  may  be  prohibitive,  and 
elimination  of  all  hazards  may  require  too  severe  a 
performance  penalty. 

2.  Hazard  Prevention  vs.  Hazard  Control 

Risk  is  reduced  by  reducing  hazard  likelihood  or 
severity  or  both.  Hazards  can  be  prevented,  or  they 
can  be  detected  and  treated.  Prevention  of  hazards 
tends  to  involve  reducing  functionality  or  inhibiting 
design  freedom,  while  detection  of  hazards  is  diffi¬ 
cult  and  unreliable. 

3.  Hazard  Prevention 


The  goal  is  intrinsic  safety  through  design  to  make 
software  faults  and  failures  non-hazardous.  General 
techniques  include: 

•  minimization  of  complexity 

•  separation  of  safety-critical  functions  and  data 

•  limit  actions  of  software 

•  minimize  interfaces  (not  only  for 
reliability  enhancement,  but  to  aid  in 

*  verification  and  certification  of  safety) 

•  firewalls 

•  authority  limitation,  access  limitations 

•  minimization  of  hazardous  states  or  time  in 
hazardous  states 

•  control  flow  limitations,  sequence  control 
(e.g.,  concurrency  and  synchronization, 
batons,  handshaking) 

•  protection  against  credible  hardware  failures 

•  hierarchical  design. 

4.  Hazard  Detection  and  Treatment 

a.  Detection  of  unsafe  states 

The  first  step  is  identification  of  safety-critical 
items.  General  techniques  include:  assertions, 
monitoring,  exception-handling,  watchdog  timers, 
acceptance  tests,  algorithm  redundancy,  and 
voting.  Types  of  checks  include:  replication 
checks,  timing  checks,  reversal  checks,  coding 
checks,  reasonableness  checks,  structural  checks, 
diagnostic  checks,  and  checks  fo.  hazardous  con¬ 
ditions. 

b.  Recovery 

Differentiate  between  fail-operational,  fail-soft, 
and  fail-safe  behavior;  backward  vs.  forward 
recovery.  Discuss  masking  through  redundancy 
and  ”Oting  (n-version  programming),  backward 
recovery  (recovery  blocks),  forward  recovery 
(robust  data  structures,  dynamic  alteration  of  flow 
of  control,  reconfiguration,  ignoring  single  cycle 
errors,  reduced  function  multiple  control  modes, 
designing  for  a  safe  side). 

VI.  Verification  and  Certification  of  Safety 

Differentiate  between  verification,  validation,  and  cer¬ 
tification  in  terms  of  product  and  process.  Describe 
and  give  examples  of  SFTA,  Software  Common  Mode 
Analysis,  SNSA. 

VII.  Assessment  of  Safety 

1.  Introduction  to  Quantitative  Risk  Assessment 

Present  a  general  introduction  to  quantitative  risk 
analysis  [Morgan81a,  Morgan81bj;  single-valued 
best  estimate,  probabilistic,  bounding.  Discuss  the 
pros  and  cons  of  assessment  (caveats).  As  an  ex¬ 
ample,  it  might  be  interesting  to  include  a  discussion 
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of  WASH  1400  and  the  pro  and  con  discussion 
raised  by  it.  Discjss  the  Titanic  effect. 

2.  Probabilistic  Risk  Assessment 

Present  probabilistic  use  of  fault  trees  and  event 
trees,  minimal  cut  sets,  evaluation  of  boolean  ex¬ 
pressions  [Vese!y81], 

3.  Software  Reliability  and  Safety 

Describe  the  state  of  the  art  in  software  reliability 
and  safety  models  and  general  principles  of  software 
reliability  modeling  along  with  weaknesses  of 
metrics  and  reliability  growth  models  [Littlewood80, 
Kopetz79).  Discuss  the  general  approaches  that  have 
been  proposed  for  adding  cost  into  these  models 
[Arlat85,  Friedman86,  Cheung80], 

VIII.  Man/Machine  Interface  Considerations 

Software  engineers  need  to  interact  with  human  factors 
experts  and  thus  should  understand  some  of  the  impor¬ 
tant  issues  to  be  resolved  with  respect  to  human  factors 
and  software  requirements. 

Discuss  the  issues  in  allocation  of  tasks  between  man 
and  machine:  complementary  vs.  incompatible  tasks, 
static  allocation  vs.  dynamic  allocation,  the  human  as 
monitor  vs.  controller,  complacency  and  situational 
awareness;  issues  in  selecting  amount  and  type  of  in¬ 
formation  to  give  human  under  normal  and  emergency 
conditions;  maintaining  human  confidence  in  the  sys¬ 
tem. 

IX.  Miscellaneous  Issues 

Discuss  social  issues:  e.g.  what  systems  should  be 
built  using  computers;  regulation  (how?  does  the  gov¬ 
ernment  have  the  right  to  regulate?);  ethical  and  moral 
considerations  for  those  who  must  build  potentially 
dangerous  systems;  liability  issues. 
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Teaching  Considerations 


Suggested  Schedules _ 

The  material  in  the  module  can  be  taught  in  different 
ways,  depending  on  the  time  available.  The  follow¬ 
ing  examples  show  the  number  of  hours  that  might 
be  spent  on  a  short  course  (first  number  in  the 
parentheses)  or  a  full  length  course  (second  number 
in  the  parentheses).  These  numbers  are  based  on  a 
tot"’  course  length  of  10  to  30  hours.  Shorter  pres¬ 
entations  could  be  derived  by  leav  _ng  out  topics  and 
longer  ones  by  presenting  more  in-depth  material. 

I.  What  is  Software  Safety  (1-3) 

II.  Introduction  to  System  Safety  (1-2) 

III.  Management  of  Safety-Critical  Software  Projects 
(.5-2) 

IV.  Software  Safety  Modeling  and  Analysis  (2  -  4) 

V.  Software  Design  Concepts  to  Enhance  Safety  (2  - 
4) 

VI.  Verification  and  Certification  of  Safety  (2  -  4) 

VII.  Assessment  of  Safety  (.5  -  5) 

VIII.  Man/Machine  Interface  (.5  -  2) 

IX.  Miscellaneous  Issues  (.5  -  4) 


the  introduction  of  a  robot  into  such  a  setting.  The 
overall  safety  requirements  would  be  Asimov’s 
Three  Laws  of  Robotics  [Asimov84j: 

1. A  robot  may  not  injure  a  human  being,  or 
through  inaction,  allow  a  human  being  to 
come  to  harm. 

2.  A  robot  must  obey  the  orders  given  to  it  by 
human  beings  except  where  i,uch  orders 
would  conflict  with  the  First  Law. 

3.  A  robot  must  protect  its  own  existence  as 
long  as  such  protection  docs  not  conflict 
with  the  First  or  Second  Laws. 

Using  these  high-level  requirements,  the  student 
must  write  system  and  software  safety  requirements 
for  the  robot.  The  results  might  be  used  by  the  class 
in  a  mock  safety  review. 

Another  type  of  exercise  might  involve  the  students 
actually  performing  a  software  fault  tree  analysis  on 
a  piece  of  code. 


Support  Materials 


It  is  recognized  that  teaching  this  material  will  re¬ 
quire  ample  support  materials.  There  is  a  survey 
paper  available  now  to  be  used  with  a  short  version 
of  the  course  and  a  book  in  preparation  for  a  more 
in-depth  course,  fhe  support  materials  package  for 
the  module  will  eventually  include  sample  safety 
plans,  sample  safety  requirements,  sample  system 
and  software  hazard  analyses,  a  fault  tree  analysis, 
etc. 


Exercises 


Exercises  and  discussion  questions  are  provided  in 
the  annotated  outline  for  the  course.  In  addition,  a 
semester  project  might  involve  the  students  prepar¬ 
ing  a  preliminary  hazard  analysis  and  fault  tree  for 
some  common  activity  with  which  the  student  is 
familiar.  The  second  step  would  be  to  hypothesize 
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this  is  a  relatively  new  software  research  area,  em¬ 
phasis  is  placed  on  delineating  the  outstanding  is¬ 
sues  and  research  topics. 

There  is  a  very  long  bibliography  at  the  end  to  aid 
in  finding  further  information. 

Llttlewood80 

Littlewood,  B.  Theories  of  software  reliability:  How 
good  are  they  and  how  can  they  be  improved.  IEEE 
Trans.  Software  Eng.  SE-6,  5  (Sept.  1980),  489-500. 

Abstract:  An  examination  of  the  assumptions  used 
in  early  bug-counting  models  of  software  reliability 
shows  them  to  be  deficient.  Suggestions  are  made 
to  improve  modeling  assumptions  and  examples  are 


given  of  mathematical  implementations.  Model  ver¬ 
ification  via  real-life  data  is  discussed  and  mini¬ 
mum  requirements  are  presented.  An  example 
shows  how  these  requirements  may  be  satisfied  in 
practice.  It  is  suggested  that  current  theories  are 
only  the  fir st  step  along  what  threatens  to  be  a  long 
road. 

An  explanation  and  evaluation  of  software 
reliability  measurement. 

Morgan8la 

Morgan,  M.  G.  Probing  the  Question  of  Technology- 
Induced  Risk.  IEEE  Spectrum  18,  11  (Nov.  1981), 
58-64. 

Abstract:  In  the  first  of  two  articles  on  risk  assess¬ 
ment  and  management,  the  author  explores  what 
constitutes  risk.  A  second  article  will  examine 
questions  of  regulation  and  management  of  risk. 

An  introduction  to  the  problems  and  techniques  in¬ 
volved  in  risk  assessment. 

Morgan81b 

Morgan,  M.  G.  Choosing  and  Managing 
Technology- Induced  Risk.  IEEE  Spectrum  18,  12 
(Dec.  1981),  53-60. 

Abstract:  This  is  the  second  of  two  articles  on  risk 
assessment.  In  the  first,  the  author  developed  a 
framework  for  thinking  about  risk.  In  this 
framework,  two  processes — exposures  and  effects — 
act  upon  natural  processes  and  human  activities  to 
produce  some  effect  or  change.  The  other  two  proc¬ 
esses — perception  and  evaluation — act  upon  this 
change  and  develop  some  notion  of  risk  through 
costs  and  benefits,  and  this  is  developed  upon. 

Neumann85 

Neumann,  P.  G.  Some  Computer-Related  Disasters 
and  Other  Egregious  Horrors.  ACM  Software  Engi¬ 
neering  Notes  10, 1  (Jan.  1985),  6-7. 

Perrow84 

Perrow,  C.  Normal  Accidents:  Living  with  High 
Risk  Technologies.  Basic  Books,  New  York,  1984. 

This  book  does  not  really  deal  with  computers,  but 
it  is  an  excellent  study  of  the  factors  involved  in 
accidents  in  complex  systems.  It  includes  details  of 
many  accidents  in  various  application  areas  includ¬ 
ing  nuclear  power,  transportation,  manufacturing, 
energy,  and  defense.  It  also  discusses  the  causes  of 
accidents  from  an  organizational  standpoint.  It  is 
extremely  interesting  and  easy  to  read. 
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